Skip to content

feat: add optimized chunking strategies for MSSQL#933

Open
vaibhav-datazip wants to merge 12 commits into
stagingfrom
feat/optimized-mssql-chunking
Open

feat: add optimized chunking strategies for MSSQL#933
vaibhav-datazip wants to merge 12 commits into
stagingfrom
feat/optimized-mssql-chunking

Conversation

@vaibhav-datazip
Copy link
Copy Markdown
Collaborator

@vaibhav-datazip vaibhav-datazip commented Apr 30, 2026

Description

  • IAM walk chunk planning: Adds a fast physloc-boundary planner using sys.dm_db_database_page_allocations (LIMITED) to stream (file_id, page_id) and generate page-aligned %%physloc%% chunks without scanning/sorting the table; used when supported (SQL Server 2012+ + VIEW DATABASE STATE, not Azure SQL DB/MI).

  • Sampling fallback + shared abstraction: When IAM walk is unavailable/fails, adds TABLESAMPLE SYSTEM + %%physloc%% sampling to estimate evenly-spaced boundaries using only SELECT permission; the sampling % calculation and clamp constants are shared across MSSQL + Oracle.

Dependencies / requirements:

  • SQL Server 2012+, VIEW DATABASE STATE permission, and not Azure SQL DB/MI for IAM-walk planning.
  • Existing strategies remain as fallback when IAM-walk is unavailable.

Fixes # (issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • Scenario A: MSSQL sync with IAM-walk enabled
  • Scenario B: IAM-walk unsupported (capability/permission/platform) falls back cleanly
    • Remove/deny VIEW DATABASE STATE (or run on Azure SQL DB/MI) and re-run the same sync.
    • Verify logs show fallback to existing strategy and sync proceeds.

Screenshots or Recordings

Documentation

  • Documentation Link: [link to README, olake.io/docs, or olake-docs]
  • N/A (bug fix, refactor, or test changes only)

Related PR's (If Any):

@vishalm0509 vishalm0509 requested a deployment to integration_tests May 6, 2026 11:33 — with GitHub Actions Waiting
Copy link
Copy Markdown
Collaborator

@vishalm0509 vishalm0509 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Image
  2. Please change PR title to that of MSSQL

Comment thread pkg/jdbc/jdbc.go
Comment thread drivers/mssql/internal/backfill.go Outdated
Comment thread drivers/mssql/internal/backfill.go Outdated
Comment thread drivers/mssql/internal/backfill.go Outdated
Comment thread drivers/mssql/internal/backfill.go
Comment thread drivers/mssql/internal/backfill.go
Comment thread drivers/mssql/internal/backfill.go
Comment thread drivers/mssql/internal/backfill.go Outdated
Comment thread drivers/mssql/internal/backfill.go
Comment thread drivers/mssql/internal/backfill.go
@vaibhav-datazip vaibhav-datazip changed the title feat: add optimized chunking strategies feat: add optimized chunking strategies for MSSQL May 12, 2026
Comment thread drivers/mssql/internal/backfill.go
Comment thread drivers/mssql/internal/backfill.go Outdated
Comment thread drivers/mssql/internal/backfill.go Outdated
Comment thread drivers/mssql/internal/backfill.go
Comment thread drivers/mssql/internal/backfill.go
Comment thread drivers/mssql/internal/backfill.go Outdated
Comment thread drivers/mssql/internal/backfill.go Outdated
Comment thread drivers/mssql/internal/backfill.go Outdated
Comment thread drivers/mssql/internal/backfill.go Outdated
Comment thread drivers/mssql/internal/backfill.go
Copy link
Copy Markdown
Collaborator

@vishalm0509 vishalm0509 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread drivers/mssql/internal/backfill.go
Comment thread pkg/jdbc/jdbc.go
Comment thread drivers/abstract/backfill.go Outdated
Comment thread drivers/mssql/internal/backfill.go
Comment thread drivers/mssql/internal/backfill.go Outdated
@vikaxsh
Copy link
Copy Markdown
Collaborator

vikaxsh commented May 20, 2026

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants